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ABSTRACT 

The  next  generation  Soldier  will  be  required  to  provide  control  for  both  ground  and  air  unmanned  systems  in  support  of 
future  combat  operations.  Unmanned  systems  show  a  great  deal  of  promise  in  that  they  will  increase  Soldier  safety  and 
enhance  situational  awareness,  but  based  on  their  current  state  of  autonomy,  will  require  high  levels  of  interaction  from 
the  operator.  These  interactions  will  range  from  platform  and  payload  control  while  mounted  within  a  combat  vehicle 
to  dismounted  outside  or  some  distance  away  from  the  vehicle,  and  will  be  performed  in  addition  to  his  or  her  primary 
mission.  In  order  to  effectively  work  together  with  these  systems,  he  or  she  must  have  a  consistent  interface  that 
provides  intuitive  control  of  primary  unmanned  system  functions  and  does  not  impose  a  unique  training  burden.  In 
addition,  the  interface  must  be  able  to  present  information  to  the  Soldier  independent  of  display  size  and  environmental 
conditions,  minimize  power  and  weight,  and  provide  the  proper  control  devices  necessary  to  manipulate  vehicle 
functions  to  include  mobility,  sensors  and  weapons. 
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1.  INTRODUCTION 


At 


The  Technologies  for  Human-Robot  Interactions  (HRI)  in  Soldier-Robot  Teaming  Army  Technology  Objective  (ATO) 
was  established  in  2004  as  a  joint  program  between  the  Tank-Automotive  Research,  Development  and  Engineering 
Center  (TARDEC)  and  the  Army  Research  Laboratory’s  (ARL)  Human  Research  and  Engineering  Directorate  (HRED), 
both  agencies  under  the  Army  Materiel  Command’s  (AMC)  Research,  Development  and  Engineering  Command 

(RDECOM).  As  of  Fiscal  Year 
2006,  HRI  has  been  combined 
with  RDECOM ’s  Aviation  and 
Missile  Research,  Development 
and  Engineering  Center’s 
(AMRDEC)  Unmanned 

Autonomous  Collaborative 
Operations  (UACO)  ATO  to 
become  the  Robotics 

Collaborations  ATO  (Fig  1). 

The  HRI  Program  came  into 
being  as  a  direct  result  of  an 
Army  Science  Board  Study1 
conducted  in  2002  that  assessed 
research  and  development 
efforts  focused  on  human- 
machine  interfaces  and 

command  and  control  of  robots 
and  found  an  absence  of  a 


Figure  1 :  Robotics  Collaboration  Army  Technology  Objective  (ATO) 
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systematic  study  of  human-robot  interface  design  exists  as  well  as  a  disconnect  between  end-users  and  the  development 
process.  Further,  they  recommended  the  establishment  of  a  Science  and  Technology  program  aimed  at  developing  a 
technical  architecture  for  human-robot  interactions  focused  on  ground  robots... the  HRI  Program.  In  addition  to  the 
development  of  a  technical  architecture,  this  program’s  objective  is  to  provide  intelligent,  scalable  mounted  and 
dismounted  control  of  ground  and  air  unmanned  systems.  This  is  accomplished  through  the  maturation  of  six  key 
pacing  technologies  and  concepts:  extensive  task  decomposition  and  analysis  to  establish  the  workload  of  the  Soldier; 
intelligent  agent  development  for  offloading  or  automating  Soldier  tasks  for  optimum  workload;  scalable  interfaces  for 
presenting  all  relevant  information  to  the  mounted  and  dismounted  Soldier  regardless  of  his  equipment  configuration 
and  mission;  recursive  modeling  to  refine  and  validate  models  developed  throughout  the  program;  exploration  of 
teaming  concepts  to  optimize  how  human  and  robot  teams  collaborate  to  perform  a  mission,  including  safe  operations  of 
unmanned  systems  amongst  humans  and  improving  local  security  and  situational  awareness  around  the  vehicle  due  to 
limit  or  constrained  sensor  views. 

With  the  addition  of  AMRDEC’s  UACO  Program,  this  ATO  has  enlarged  to  accommodate  collaborative  behaviors 
between  unmanned  air  systems,  which  further  reduces  the  workload  on  the  Soldier.  This  paper,  however,  will  focus 
primarily  on  the  Human-Robot  Interactions  portion  of  this  Army  Technology  Objective. 

The  intent  of  this  paper  is  to  provide  background  information  on  the  HRI  Program,  highlight  the  focus  of  its  use  within 
the  Army’s  Future  Force,  explain  concepts  of  scalability  for  extending  unmanned  system’s  control  from  mounted  to 
dismounted  operations,  and  provide  an  overview  of  technologies  under  investigation  for  use  in  this  program. 

Section  two  of  this  paper  provides  details  of  the  intended  application  of  the  technology  developed  under  the  HRI 
Program.  Specifically,  it  address  platoon  and  lower  operations  of  the  future  force  where  utilization  of  unmanned 
systems  is  identified,  and  does  not  cover  special  applications  of  robot  use.  Products  from  this  program  primarily  feed 
Future  Combat  Systems’  concepts  of  operations. 

Section  three  outlines  the  approach  the  HRI  Program  will  utilize,  to  include  architecture,  automations  and  presentation, 
to  address  scaling  functions  across  varying  display  sizes  and  missions.  Questions  such  as  how  is  the  same  look  and  feel 
for  the  execution  of  a  task  maintained,  especially  with  less  display  space  for  dismounted  operations  and  how  does  the 
Soldier  still  perform  his/her  primary  tasks  with  the  added  responsibility  of  unmanned  systems  control  are  addressed. 

Section  four  provides  an  assessment  of  current  technologies  under  investigation  for  incorporation  or  leveraging  within 
the  HRI  Program.  Primarily  display  technologies  are  explored,  as  well  as  inceptor  devices  for  interacting  with  the 
systems  under  control. 

Section  five  of  this  paper  concludes  with  future  plans  for  the  program,  and  identifies  areas  of  opportunity  in  the  field  of 
human-robot  interactions. 


2.  APPLICATION  FOCUS  FOR  HRI 

The  U.S.  Army  is  in  the  process  of  a  major  transformation;  it  is  becoming  more  agile  and  modular  in  structure,  which 
will  permit  it  to  see  first,  understand  first,  decide  and  act  first,  finish  decisively,  and  survive  and  endure.  This 
capabilities-based  Force  will  be  more  responsive,  faster,  have  extended  reach  and  increased  effectiveness.  In  addition, 
this  highly  flexible,  networked  force  will  be  able  to  rapidly  deploy  and  task  organize  to  deal  with  contingencies  in  joint 
warfare  and  conflict  resolution.  The  Future  Combat  Systems  force  structure  is  shown  below  (Fig  2),  and  supports 
operations  in  the  2010  timeframe  and  beyond.  Part  of  this  force  structure  change  includes  the  organic  employment  of 
unmanned  systems,  which  will  increase  both  the  standoff  distance  and  time,  keep  Soldiers  out  of  harm’s  way,  and 
provide  increased  situational  awareness  to  the  Soldier. .  .extending  the  capability  of  military  operations.  The  majority 
of  robotic  control  will  be  conducted  at  the  squad  through  company  level,  with  the  exception  of  Class  III  and  Class  IV 
Unmanned  Aerial  Vehicles  (UAV’s),  which  are  deployed  at  the  battalion  and  brigade  echelon’s,  respectively.  The  HRI 
Program  will  focus  on  platoon  and  below  operations  and  the  Manned  Ground  Vehicle  (MGV)  variants  that  provide 
unmanned  systems  control.  The  appropriate  vehicles  are  highlighted  in  red  in  Figure  2,  and  primarily  consist  of  the 
following  MGV  variants:  Command  and  Control  Vehicle  (C2V),  the  Reconnaissance  and  Surveillance  Vehicle 


(R&SV),  Mounted  Combat  System  (MCS)  and  Infantry  Carrier  Vehicle  (ICV).  The  unmanned  systems  focus  is  on  the 
Armed  Robotics  Vehicle  (ARV,  all  variations),  the  Multifunction  Utility/Logistics  Equipment  (MULE,  all  variations), 
Unattended  Ground  Sensors  (UGS)  and  Class  I  and  II  UAV’s.  Figure  3  depicts  the  vehicle  systems  of  FCS,  with  the 
manned  systems  on  the  left,  the  unmanned  systems  on  the  right  and  the  Soldier/Network  in  the  middle,  tying  the 
systems  together.  K&V 
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behaviors  and  autonomy  that  will  make  UAV’s  more  capable  and  reduce  the  controlling  burden  on  the  Soldier.  Also, 
under  the  Near-Autonomous  Unmanned  Systems  ATO,  TARDEC’s  ARV  Robotic  Technologies  Program  is  developing 
tactical  behaviors  for  ARV  platforms  that  will  provide  these  unmanned  systems  with  autonomous  capability,  as  well  as 
exploring  vehicle  self-defense  to  include  human  intent  analysis,  and  integration  of  advanced  perception  capabilities  to 
provide  the  Soldier  with  enhanced  situational  awareness.  Together,  these  Army  programs  are  advancing  the  state  of  the 

art  for  robotic-related 
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technologies. 

The  next  section  explains  the 
methodology  the  HRI  Program 
will  employ  to  tackle  the  issues 
of  portability  and  distributed 
knowledge  through 

architecture.  Soldier  task 
automation  to  reduce  workload 
and  increase  situational 
awareness,  and  scalability  of 
the  interface  to  provide  a 
constant  look  and  feel  for  like 
tasks  across  varying  hardware 
configurations.  A  control 
architecture  provides  the 
framework  for  implementation 
of  services  and  data  sharing, 
enabling  easy  integration  and 
plug  and  play  capability  of 
mature  software  and  hardware 
products.  On  this  baseline 


Figure  3:  FCS  Vehicle  Systems 


technique,  termed  presentation  description  document  (PDD),  has  been  identified  that  provides  for  the  specification  of 
the  FCS  WMI  in  separable  meta  format  files  which  can  be  assimilated  and  executed  at  run  time.  The  HRI  Program  has 
defined  a  similar  approach  with  respect  to  meta  graphical  interface  description;  however,  the  HRI  Program  is 
additionally  addressing  WMI  scalability  and  adaptability  and  as  such,  has  incorporated  metadata  above  and  beyond  that 
currently  under  development  within  the  FCS  program.  As  with  the  service  based  approach;  however,  the  intent  is  to 
provide  a  WMI  description  technique  that  can  be  readily  correlated  to  the  FCS  PDD  and  to  eventually  provide  the 
ability  to  convert  PDD’s  to  HRI  scalable  interface  description  and  vice-versa. 

To  enable  a  more  direct  association  with  the  FCS  program,  the  HRI  Program  is  employing  the  FCS  System  of  Systems 
Common  Operating  Environment  (SOSCOE)  as  a  middleware  service  layer,  which  will  be  leveraged  primarily  for  its 
communication  (discovery  and  dissemination),  repository  (data  store),  configuration  and  control,  interoperability,  and 
task  integration  network  services.  This  will  facilitate  components  designed  within  the  HRI  Program,  although 
implemented  differently,  to  be  correlated  to  and  in  some  cases  interfaced  with  FCS  manned  ground  vehicle  and 
common  crew  workstations,  dismounted  warrior  and  control  device  systems,  and  system  of  systems  integration 
laboratory  assets. 

3.1.2.  Scalable  Interface 

Central  to  the  control  and  interaction  with  unmanned  systems,  is  the  WMI.  Within  the  HRI  Program,  the  WMI  provides 
for  all  interactions  with  the  unmanned  system  and  is  not  bounded  by  today’s  control  concepts,  typically  implemented 
via  graphical  user  interfaces  packaged  into  operator  control  units.  Driven  by  the  necessity  to  advance  operator 
effectiveness  in  multi-tasking  environments,  the  HRI  Program  is  researching  and  analyzing  various  unmanned  system 
control  and  interaction  techniques  to  include  haptic,  tactile,  3D  audio,  and  speech  recognition.  Given  these  focus  areas, 
the  WMI  must  be  flexible  enough  to  accommodate  technology  insertion  and  non-traditional  input/output  techniques 
along  a  non-linear  traditional  GUI  continuum.  An  example  might  be  controlling  a  robot  via  joystick  (tele-op)  on  one 
end  of  the  continuum,  while  controlling  a  robot  via  hand  gestures  and/or  speech  at  the  other  extreme.  To  facilitate 
technology  analysis  and  experimentation  along  this  continuum,  the  HRI  Program  is  developing  a  scalable  architecture 
with  the  high  level  goals  of  supporting  commonality,  scalability,  portability,  and  adaptability. 

In  order  to  further  describe  the  techniques  employed  within  the  scalable  interface  approach,  it  is  important  to 
differentiate  scalability  as  defined  by  the  HRI  Program  from  the  traditional  interpretation  of  shrinking  or  stretching  a 
graphical  interface  as  the  WMI  is  ported  among  displays  of  varying  size.  For  HRI,  scalability  is  a  farther  reaching 
concept,  addressing  WMI  scalability  that  includes  graphical  stretching/shrinking,  but  additionally  takes  into  account 
configuration,  alternate  graphical  representation,  role-based  representation,  and  prioritization  of  interface  information. 

The  concept  presented  in  Figure  6  below  depicts  the  specification  approach  to  support  HRI  scalability.  The  WMI 
description  is  encapsulated  in  four  files.  These  are  the  interface  description,  which  defines  the  graphical  user  interface 
layout  in  a  non-device/graphics  independent  manner;  the  destination  hardware  description,  which  defines  the  target 
hardware  environment  to  which  the  WMI  will  be  deployed;  the  user  preferences,  which  defines  the  style  for  the 
resultant  rendering  of  the  graphical  objects;  and  finally  HFE  guidelines,  which  provides  human  factors  information  and 
rules  that  will  be  combined  with  the  style  and  interface  to  deploy  the  WMI  to  the  specified  target  display  The  scalable 
interface  description  information  is  processed  by  the  scaling  engine,  which  in  turn  generates  a  widget  display  list.  The 
widget  display  list  is  processed  by  a  conceptual  presenter  and  rendered/translated  to  the  target  system.  In  some  cases, 
the  target  system  could  simply  be  a  display.  Alternately,  the  target  system  could  be  the  combination  of  a  display  with  a 
speech  recognition  system,  or  an  HMD  with  a  tactile  control/feedback  system.  The  intended  goal  scalable  interface  for 
the  HRI  Program  is  to  be  able  to  represent  one  interface  description  on  any  of  these  combinations.  Within  the  HRI 
Program  these  scalable  interface  descriptions  will  be  utilized  to  specify  and  interchange  human-robotic  information 
between  robotic  vehicles  and  controllers,  supporting  discovery  and  reconfiguration  of  robotic  systems. 


As  described  in  section  2,  the  FCS  program  provides  for  the  control  of  robotic  assets  via  its  manned  ground  vehicle 
systems.  This  control  is  implemented  via  a  set  of  battle  command  services  interfaced  through  operator  workstations  and 
MGV  common  crewstations.  Control  of  FCS  unmanned  assets  from  the  dismounted  operator  is  provided  through 
dismounted  control  and  Soldier-borne  devices.  Battle  command  services  are  coordinated  through  task  integration 
networks  (TIN’s)  which  can  be  interfaced  to  the  operator  via  the  Warfighter-Machine  Interface  (WMI).  The  WMI,  in 
turn,  is  described  via  presentation  description  documents  (PDD)  that  are  processed,  displayed  to,  and  interacted  with  by 
the  crewstation  operator.  Currently  UAV’s  are  targeted  for  integration  within  FCS  spin  out  2;  to  be  fielded  in  FY12, 

while  UGV’s  are  targeted  for  integration 
within  FCS  spin  out  3;  to  be  fielded  FY14 
(See  Figure  5). 

At  this  point  in  time,  the  FCS  manned  and 
unmanned  vehicle  architectures,  WMI,  and 
battle  command  services  are  still  evolving 
and  will  be  made  available  through  a 
coordinated  build  plan  approach  supporting 
the  development  of  spin  out  2  and  spin  out  3 
infrastructure  and  services. 

Given  that  the  HRI  products  and 
experimental  data  in  their  final  form  are 
focused  towards  transition  to  and  application 
within  the  FCS  program,  an  architectural 
approach,  further  described  in  the  following 
subsection,  has  been  defined  that  provides  a 
parallel  to  the  FCS  architecture  with  respect 
to  WMI,  agent  insertion,  and  task  network 
integration  to  more  easily  allow  for  transition 

Figure  5:  FCS  Spin  Out  Architecture  throughout  the  FCS  program  SDD  and  spin 

out  increment  phases. 


3.1.1.  Architecture 

While  the  task  of  human/robotic  interface  analysis  is  focused  primarily  on  the  manner  in  which  the  human  operator 
interacts  to  control  and  monitor  one  or  more  robotic  assets,  it  is  important  to  remember  that  this  interaction  reaches  far 
beyond  traditional  operator  control  units,  displays,  and  input  devices.  Essential  to  the  analysis  is  the  integration  of 
automation  and  agents  to  assist  the  operator  in  control  and  monitoring  (integrated  within  both  the  manned  and 
unmanned  platforms)  as  is  also  the  data  model  that  enables  agents,  operators,  and  WMI’s  to  interface  with  robotic 
entities. 

For  the  HRI  Program,  it  was  desirable  to  develop  an  approach  that  would  allow  not  only  the  advancement  of  research  in 
key  program  areas,  but  at  the  same  time  provide  synergy  and  data  transition  with  the  FCS  program.  However,  the  HRI 
Program  was  at  a  bit  of  a  disadvantage  towards  meeting  this  specific  objective,  given  that  the  development  of  WMI  and 
Crewstation  components  for  the  FCS  program  are  still  in  requirements  and  early  design  phases.  To  overcome  this 
obstacle,  the  HRI  Program  analyzed  relevant  FCS  program  areas  (common  crewstation,  dismounted  control  devices, 
battle  command,  and  WMI  services)  and  evolved  an  approach  that  would  provide  for  correlation  among  FCS  and  HRI 
technologies/components.  The  approach  centered  around  two  main  elements.  The  first  of  these  elements  was  the 
specification  of  a  service  based  architecture  approach,  providing  for  the  discovery  of  and  interaction  with  services, 
agents,  and  automations.  Analysis  and  task  decomposition  conducted  within  the  analysis  phase  of  the  HRI  Program 
will  be  utilized  to  derive  the  services  as  well  as  the  interconnecting  logical  data  model.  The  resultant  task  networks, 
services,  and  data  model  will  be  developed  in  a  manner  that  facilitates  correlation  to  FCS  battle  command  services.  The 
second  element  was  the  definition  of  an  WMI  specification  technique.  Within  the  FCS  program  a  specification 


capability,  task  automations  that  reduce  workload  when  the  Soldier  is  overloaded  or  stressed  can  first  be  identified,  then 
developed  and  readily  integrated  into  the  system,  then  simulated  to  explore  the  overall  increase  in  performance.  These 
automations  are  complimented  by  smart  Soldier-Machine  Interface  (SMI)  configurations  and  layouts  that  make  the 
system  easier  to  use.  Lastly,  scaling  tasks  so  that  the  same  functionality  can  be  represented  on  varying  hardware 
configurations  is  explored.  The  main  point  here  is  that  as  display  space  is  lost  when  going  from  a  mounted  mission  with 
multiple  large  displays  down  to  perhaps  a  wearable  display  when  performing  dismounted  operations,  how  are  tasks 
performed  through  a  combination  of  visual  representation  and  alternative  devices  that  make  up  for  lost  display  real 
estate. 


3.  PROGRAM  METHODOLOGY 

TARDEC  and  ARL-HRED  are  teaming  to  solve  human-robot  interactions  for  mounted  and  dismounted  control  of 
ground  and  air  unmanned  systems.  An  end-to-end  systems  engineering  approach  has  been  employed  to  solve  this 
problem;  starting  with  the  identification  of  soldier  requirements  for  conducting  missions  while  teaming  with  unmanned 
assets.  Requirements  are  decomposed  from  system  level  behaviors  through  the  identification  of  operator  roles  and  tasks 
required  to  implement  those  roles  within  specific  use  cases  or  operational  contexts  (i.e.,  mission  threads).  Identified 
tasks  are  further  decomposed  into  task  networks  captured  via  an  intelligent  systems  ontology  that  defines  both  control 
and  data  flow  within  the  networks.  To  the  extent  possible,  task  networks  are  derived  independent  of  specific  technology 
or  allocation  (e.g.,  hardware,  architecture,  or  human  operators).  As  behaviors  are  derived  and  captured,  modeling  and 
analysis  is  conducted  through  the  mapping  of  task  networks  to  roles,  which  are  then  deployed  to  system  level 
components  capable  of  executing  the  defined  roles.  These  deployments  are  then  constructively  simulated  utilizing  an 
intelligent  systems  behavior  simulator  (depicted  below  in  Figure  4). 

As  represented  in  the 
figure,  the  behavior 
simulator  executes  task 
networks  that  have  been 
specified  and  captured 
within  the  knowledgebase 
during  model  development 
and  translated  into 
executable  form  during  load 
time  processing. 

Intermediate  data  produced 
in  the  load  time  processing 
will  be  generated  (as 
required  by  the  models)  to 
support  model  interfacing 
to  external  agents  and 
simulation/stimulation 

environments.  These  interactions  dynamically  stimulate  the  task  networks  during  run  time,  which  in  turn  provides  the 
capability  to  measure  the  effectiveness/performance  of  the  deployment  for  a  specific  use  case  or  environmental 
scenario.  In  general,  the  HRI  Program  is  focused  on  the  variation  of  deployment  technologies  (displays,  input 
devices/ techniques,  and  agents)  within  this  phase  of  the  program  to  isolate  viable  candidates  for  system  integration  and 
field  test.  The  primary  metrics  utilized  within  this  phase  of  the  analysis  are  operator  workload  and  effectiveness  with 
respect  to  human/robotic  task  allocation. 

3.1.  Program  Components 

For  validation  and  experimentation  purposes,  candidate  HRI  technologies  identified  within  the  analysis  phase  of  the 
program  will  be  further  developed  and  integrated  into  mounted  and  dismounted  configurations  and  tested  within 
crewstation  and  Soldier  testbeds  in  the  HRI  system  integration  laboratory  (SIL).  Maturation  and  validation  of  these 
technologies  within  the  SIL  environment  will  in  turn  lead  to  refinement  and  eventual  system  field  test  and 
experimentation,  supporting  future  force  operations  and  the  FCS  program. 


Figure  4:  ISBS  Behavior  Simulator 
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Figure  6:  Scalable  Interface  Description  Approach 


3.1.3.  WMI  Guidelines 

An  essential  component  supporting  the  standardization  of  common  WMI  is  the  establishment  of  a  set  of  guidelines 
governing  WMI  methodology  and  principles  within  the  relevant  domain.  The  HRI  Program  is  analyzing  requirements 
in  order  to  establish  a  consistent  set  of  guidelines  that  can  be  interpreted  within  an  interface  to  support  common  look, 
feel,  and  scalability.  In  some  instances,  these  guidelines  will  be  incorporated  into  the  HFE  metadata  encoding  and  will 
address  the  following  areas  of  scalable  interface  design:  configurability,  display  layout,  data  logging,  operator  login, 
status,  and  modes  of  operation.  Configurability  addresses  dynamic  reconfigurability  of  controller  and  robot  interfaces 
providing  the  ability  to  upload  platform  configuration  specific  data  for  display.  Platform  specific  data  could  include 
mission  package  configuration,  mission  package  controls  and  status,  as  well  as  robotic  platform  specific  controls  and 
status.  Display  layout  addresses  the  manner  in  which  a  screen  can  be  physically  and/or  virtually  addressed  supporting 
the  placement  of  controls  and  status  indicators.  General  terms  and  methods  applicable  across  interfaces  would  be 
defined  in  an  abstract  manner  such  that  they  could  be  mapped  accordingly  within  a  physical  interface  design,  for 
example  warning,  caution,  and  alert  (WCA)  levels  and  help.  To  the  degree  applicable,  static  display  areas  will  be 
defined  for  the  display  of  critical  data  such  as  status  and  WCA’s.  Data  logging  addresses  the  type  of  data  that  may  be 
logged  for  training,  validation,  and/or  analysis  purposes.  Logged  data  could  include  WCA,  audio,  images,  and  session 
data.  Operator  login  addresses  the  manner  in  which  an  operator  gains  access  to  the  WMI,  systems  controlled  by  the 
interface,  and  the  role  access  of  the  operator  within  a  session.  Status  addresses  the  manner  in  which  operator  critical 
status  information  is  defined  and  accessed.  Critical  status  information  could  include  robotic  vehicle  status,  mission 
status,  vehicle  orientation,  communication  link  status,  and  battery  levels.  Modes  of  operation  address  controller  modes 
of  operation  and  the  interface  behavior  within  each  respective  mode.  Example  modes  could  include  setup 
(configuration,  diagnostics,  log/other  status),  pre/post  operational  (mission  planning  and  mission  analysis),  operational 
(tactical  operations),  sleep  (reserve),  and  training  (embedded  training,  mission  playback  and  analysis). 

4.  HARDWARE 

In  general,  controls  and  displays  used  by  the  military  can  be  put  into  three  broad  categories:  large  single  or  multi-screen 
crew  station  configurations  typically  used  in  manned  ground  vehicles;  portable  single-screen  environmentally  protected 
suitcase-sized  devices  for  use  in  a  host  of  environments  or  COTS  workstations  for  Tactical  Operations  Center  (TOC) 
type  operations;  and  small  highly  portable  and  light  weight  devices  to  be  used  by  the  dismounted  Soldier. 


4.1.  Crew  Stations 

Crew  stations  offer  the  most  display  space  and  versatility  for  control  of  manned  and  unmanned  vehicle  assets.  Because 
it  is  vehicle  based  and  nothing  is  carried  by  the  Soldier,  the  crew  station  has  a  greater  quantity  of  display  and  control 
space  maximizing  information  display  and  system  control.  Typically  currently  fielded  vehicle  crew  stations  will  have  at 
least  one  and  usually  more  displays  dedicated  to  a  specific  vehicle  function  such  as  target  acquisition  or  map  services. 
Likewise,  controls  are  dedicated  hard  buttons,  switches,  dials,  etc.  for  control  of  specific  vehicle  functions.  The  picture 
on  the  left  in  Figure  7  shows  the  gunner’s  station  of  a  Stryker  vehicle  as  an  example  of  this.  Display  screens  tend  to  be 
active  matrix  or  thin  film  transistor  based  LCD  screens  because  of  the  good  image  quality  for  displaying  map  data,  wide 
color  range,  and  thin  form  factor  saving  interior  vehicle  space. 


Figure  7:  Stryker  crewstation  (left)  &  Prototype  TARDEC  crewstation  (right) 


Future  designs  will  enlarge  displays  and  add  soft  reconfigurable  controls  minimizing  the  amount  of  subsystem  unique 
dedicated  controls  and  increasing  screen  size  for  maximum  information  display.  Figure  7  shows  a  functional  crew 
station  prototype  that  can  be  used  in  the  performance  of  multiple  mission  types  and  for  control  of  unmanned  ground  and 
air  vehicles.  Notice  how  the  amount  of  display  space  has  increased  tremendously  over  what  is  currently  fielded  in  the 
Stryker  vehicles.  It’s  envisioned  that  a  crew  station  similar  to  the  one  on  the  right  in  Figure  7  can  be  used  in  all  the  FCS 
manned  ground  vehicle  variants  regardless  of  mission  type.  The  larger  display  space  accommodates  driving  functions, 
command  and  control,  weapons  control,  and  communications  all  from  the  same  station.  This  same  crew  station  can  be 
used  in  each  FCS  vehicle  variant  eliminating  unique  hardware  and  reducing  training  requirements  across  vehicle 
variants  due  to  unique  hardware.  Add  to  this  an  SMI  with  a  common  look,  feel,  and  function,  across  all  the  vehicle 
variants,  and  the  training  burden  is  even  further  reduced.  Because  more  information  can  be  displayed  to  the  Soldier 
either  continuously  or  on  demand,  task  performance  can  be  increased  as  ease  of  information  accessibility  and  system 
control  has  been  increased.  Add  to  this  configuration  alternative  control  and  display  capabilities  such  as  3D-audio, 
speech  recognition,  touch  panels,  etc,  and  Soldier  task  performance  increases  even  further  as  he/she  can  now  perform 
tasks  in  parallel  and  have  greater  situational  awareness  than  before.  An  additional  advantage  also  lays  in  future  growth. 
As  vehicles  age  and  new  enhancement  packages  are  added,  the  user  interface  can  be  changed  completely  in  software 
with  no  physical  modifications  needed  to  the  vehicle  crew  stations  which  can  be  a  tremendous  cost  savings. 


4.2.  Workstations  and  Portable  Computers 


Another  common  configuration  of  control/display  device  used  by  the  Army  is  the  workstation,  rugged  laptop  or  suitcase 
style  computer.  Typically  used  for  command  and  control  activities  at  the  platoon  level  and  TOC,  or  for  control  of 


unmanned  systems  from  a  command  vehicle,  fixed  location  or 
sometimes  even  on  foot.  These  are  usually  single  screen  LCD 
devices  (like  those  shown  in  Figure  8)  that  provide  command  and 
control,  communications,  and  video  or  information  feedback  from 
unmanned  systems.  Most  every  UGV,  UAV,  or  UGS  has  a 
unique  proprietary  control  and  display  device  developed  by  the 
manufacturer.  As  a  result,  scores  of  different  workstation  and 
portable  computers  find  their  way  in  theater  making 
supportability  increasing  difficult  because  of  the  number  of 
systems  that  must  be  maintained,  repaired,  or  replaced.  But  as 
the  Army  moves  forward  with  the  FCS  program  and  concepts  for 


controls  and  displays  are  matured,  multiple  unmanned  systems  will  be  controlled  from  many  fewer  devices. 

43.  Small  Soldier  Portable  Devices 

The  FCS  vision  is  for  the  dismounted  Soldier  to  also  have  full  control  of  unmanned  systems  on  par  with  that  of  the 
mounted  controller.  However,  the  dismounted  Soldier  can’t  afford  to  be  burdened  with  cumbersome  and  heavy 
equipment  as  they  lack  the  room  to  pack  the  devices,  and  any  additional  weight  becomes  an  extreme  physical  burden. 
Therefore,  the  dismounted  Soldier  needs  to  be  equipped  with  much  smaller  devices  similar  to  those  shown  in  Figure  9. 


Figure  9:  Portable  Display  Devices 


Shown  here  are  a  flexible  display,  tablet  display,  and  a  PDA  style  device.  The  preferred  device  ultimately  will  be  the 
flexible  display  as  it  offers  all  the  features  of  the  standard  LCD  display  but  in  an  extremely  thin  and  lightweight  form 
factor  that  can  even  be  integrated  into  a  Soldier’s  uniform  nearly  eliminating  all  Soldier  portable  space  claims  except  for 
the  power  source.  The  Army  estimates  that  the  first  flexible  displays  will  be  ready  for  introduction  to  the  dismounted 
Soldier  in  FY08.  The  tablet  device  offers  significantly  more  screen  space  but  at  the  cost  of  added  weight  and  space. 
The  PDA  is  the  opposite,  very  lightweight  and  compact  but  with  little  screen  space. 

Another  type  of  device  that  may  be  worn  by  the  dismounted  Soldier  is  the  Helmet/Head  Mounted  Display  or  HMD. 
The  HMD  is  usually  a  monocular  or  sometimes  a  binocular  device  involving  the  projections  of  graphical  images,  or 
overlays  onto  glass,  a  mirror,  or  directly  into  a  small  LCD  that  the  Soldier  can  view.  The  glass  option  has  transparent 
properties  making  it  possible  for  the  Soldier  to  still  see  his  environment  while  viewing  images  or  overlays  without 
having  to  look  away  from  his  point  of  interest. 

4.4.  Criteria  for  Display  Systems 

Whether  large  or  small,  there  are  criteria  necessary  for  all  displays  to  have  to  meet  for  use  in  military  environments. 
These  include: 

•  High  brightness/high  contrast  for  reading  in  direct  sunlight 

•  Operable  in  extreme  temperature  ranges  for  operation  in  all  world  regions 

•  Must  be  resistant  to  vibration  and  shock  loads  in  excess  of  20g 

•  Waterproof  or  water  resistant  for  operation  in  all  weather  conditions 

•  Low  EMI  emissions  for  low  visibility  to  detection  devices 

These  are  basic  elements  that  are  called  out  in  most  MIL-STD’s  or  industry  standards  related  to  displays.  However, 
advancements  in  technology  such  as  gaming  systems  are  driving  the  hardware  in  ways  that  the  military  hasn’t 
necessarily  considered  in  the  past  and  may  become  future  criteria  for  all  displays: 

•  3D  acceleration  for  in-vehicle  training,  mission  rehearsal,  and  battlefield  visualization. 

•  Video  rendering  for  streaming  video  from  unmanned  assets. 

•  Touch  panels  (resistive,  capacitive,  IR,  or  SAW)  for  alternate  input  methods. 

3D  acceleration  and  video  rendering  capabilities  for  Army  applications  require  more  powerful  video  cards  than  ever 
before.  Large,  high  resolution  graphical  databases  for  performing  in-vehicle  embedded  training,  mission  rehearsal,  and 
battlefield  visualization  will  require  cards  that  can  render  multiple  channels  of  graphics  with  even  more  polygons  than 
today’s  fastest  gaming  based  PC’s.  Likewise  multiple  channels  of  high  resolution  video  that  can  be  manipulated  in  real¬ 
time  (rotated  and  resized  to  fit  where  needed  in  the  display  space)  will  require  extreme  processing.  Fortunately,  the 


gaming  industry  is  constantly  moving  in  this  direction  pushing  the  state  of  the  art.  PC's  now  utilize  multiple  graphics 
cards  with  support  for  large  polygonal  models  and  terrain.  Commercial  PC  form  factors  such  as  PCI  express  are  being 
produced  in  rugged  versions  for  military  applications. 


4.5.  Tactile  Displays 


Tactile  displays  may  be  a  useful  cueing  system  for  future  military  operations.  Tactile  displays  make  use  of  vibration  or 
pressure  based  stimulators  placed  on  or  near  the  skin.  Multiple  stimulators  placed  at  varying  locations  on  the  body  can 

enhance  situational  awareness  by  indicating  the  direction  of  an  item  of  interest.  Even 
more  information  can  be  obtained  when  varying  frequency  or  intensity  of  the  pressure 
or  vibration  stimulator.  IMPRINT  modeling  and  simulation  performed  at  a  vanety  of 
Army  and  university  labs  have  indicated  that  when  combined  with  visual  and  auditory 
cueing  systems  tactile  displays  can  have  a  positive  affect  on  situational  awareness  and 
decision  making  for  the  mounted  platoon  leader2.  Typically,  haptic/tactile  displays 
for  military  usage  take  the  form  of  a  vest  (Figure  10)  worn  by  the  Soldier.  The  vest 


has  vibro-tactile  stimulators,  pressure-tactile  stimulators,  or  a  mix  of  both  embedded 
Figure  10:  Tactile  Vest  into  it  at  varying  degrees  relative  to  some  reference  point  on  the  torso,  that  can  be 

individually  activated  to  draw  the  wearer’s  attention  to  a  location  or  item  of  interest. 
They  can  be  used  to  for  such  things  as  warnings  and  alerts,  provide  assistance  with  navigation  and  guidance,  system 
locations  within  a  vehicle,  or  even  indicate  threat  warnings  external  to  the  vehicle. 


4.6.  3D-Audio  Displays 

3D-audio  display  systems  have  great  potential  for  assisting  the  Soldier  with  situational  awareness  and  task  performance. 
3D-audio  systems  spatially  organize  sound,  making  it  seem  as  though  it’s  coming  from  behind  or  from  the  side, 
allowing  the  listener  to  more  easily  process  information.  This  is  sometimes  known  as  the  “cocktail  party”  effect.  If  five 
people  are  speaking  at  the  same  time,  it’s  easier  to  focus  on  what  one  of  them  is  saying  when  they  are  spaced  out  from 
each  other.  This  effect  is  achieved  through  software  processing  of  head  related  transfer  functions  or  HRTF's.  The 
HRTF  takes  into  consideration  the  physical  properties  of  the  outer  and  inner  ear,  and  the  time  differences  associated 
with  sound  pressure  waves  reaching  the  left  and  right  ear.  A  sound  source  coming  from  the  left  will  reach  the  left  ear 
before  the  right  ear  by  a  fraction  of  a  second.  This  perceived  time  separation  is  one  of  the  things  that  makes  a  sound 
source  actually  sound  like  it’s  coming  from  a  specific  direction.  Elaborate  HRTF’s  may  also  take  into  account  sound 
source  volume  based  on  distance  from  the  listener,  sound  wave  reflections  in  the  listening  space,  and  how  sound 
propagates  through  not  only  the  ears  but  the  head  and  torso  as  well.  This  benefits  the  Soldier  by  making  it  easier  to 
hear  voice  traffic  over  the  intercom  or  radio  system.  It  can  also  be  used  to  draw  the  Soldier’s  attention  to  a  location  or 
item  of  interest  inside  or  outside  of  the  vehicle,  and  be  used  for  warnings  and  alerts,  or  as  navigation  assistance. 

3D-audio  works  best  when  heard  through  head-phones  where  extraneous  environmental  sounds  are  muffled  out  or 
eliminated.  Audio  processors  are  needed  to  apply  the  HRTF  to  an  incoming  audio  stream  in  order  to  “spacialize”  it  as 
normal  audio  such  as  that  over  an  intercom  has  no  three  dimensional  properties  other  than  perhaps  stereo.  This  audio 
processing  hardware  and  the  HRTF’s  used  to  be  costly  to  implement,  but  with  the  advent  of  3D-audio  capabilities  in 
video/virtual  world  gaming,  basic  3D-audio  capabilities  have  become  much  more  economic.  One  additional  piece  of 
hardware  potentially  required  for  3D-audio  depending  on  the  intended  application  is  a  head  tracker.  A  head  tracker 
monitors  the  orientation  of  the  listeners  head.  When  taking  this  into  account,  3D-audio  systems  can  make  a  sound 
appear  to  emanate  from  a  fixed  location  in  space,  say  to  the  north,  regardless  of  which  way  the  listener’s  head  is 
oriented.  Without  this,  the  sound  will  move  with  the  listener’s  head  as  he/she  turns  it. 


4.7.  Interface  Scalability  and  Display  Deconfliction 

Given  the  various  current  and  future  control  and  display  systems  as  described  above,  the  notion  of  scalability  becomes 
very  important.  How  do  you  efficiently  represent  information  and  provide  dynamic  control  of  manned  and  unmanned 
systems  when  moving  from  a  large  display  space  such  as  a  crew  station  to  a  smaller  display  space  such  as  a  PDA  with 
no  degradation  in  mission  performance?  It’s  clear  that  all  control  and  display  mechanisms  on  the  larger  display 
physically  will  not  fit  on  the  smaller  display.  Given  this,  why  not  display  to  the  Soldier  at  the  top  level,  the  information 
or  control  elements  most  crucial  to  his  mission  performance  or  those  needing  access  most  frequently.  Items  of 


secondary  importance  or  not  accessed  as  frequently  can  be  buried  down  into  lower  levels  or  layers  of  the  interface. 
Also,  where  possible  automate  actions  that  don’t  necessarily  need  close  Soldier  monitoring.  For  example  it  might  be 
beneficial  to  have  a  robot  navigate  autonomously  instead  of  being  tele-operated  by  the  Soldier  so  that  he  can  use  his 
limited  display  space  for  viewing  recon  sensor  information.  Automations  can  be  turned  on  and  off  as  needed  depending 
on  the  circumstances.  The  whole  process  itself  can  be  automated  through  the  use  of  intelligent  agents.  The  agents  are 
aware  of  display  size  limitations,  the  level  of  automation  available  on  platforms,  and  what  is  important  to  show  to  the 
Soldier  at  any  given  time  for  his  mission  type.  The  agents  can  then  intelligently  decide  what  to  automate  and  when,  and 
what  to  show  to  the  Soldier  and  in  what  format  and  dynamically  make  changes  before  during  or  after  a  mission  adapting 
to  the  situation,  or  they  can  simply  act  as  a  decision  aid  providing  the  Soldier  with  guidance  on  what  to  do. 

The  need  for  scalability  of  the  Soldier-machine  interface  becomes  even  more  evident  when  other  types  of  displays  such 
as  tactile  and  3D-audio  are  taken  into  account.  The  addition  of  these  display  types  allows  much  greater  flexibility  in 
how  to  scale  or  deconflict  elements  of  the  user  interface.  Take  for  example  the  mission  of  controlling  both  movement 
and  payloads  of  two  UGV’s.  From  a  large  multi-screen  crew  station,  finding  the  display  space  to  place  all  soft  buttons, 
post  alerts  and  notifications,  and  receive  video  and  sensor  data  back  from  the  robots  is  relatively  easy.  This  is  not  the 
case  for  the  same  mission  with  only  a  single  screen  workstation  as  an  OCU.  Displaying  all  this  information  to  the 
Soldier  without  compromising  his/her  performance  or  task  overloading  him/her  may  be  difficult.  The  use  of  these  other 
devices  allows  for  the  flexibility  to  port  many  of  those  functions  off  the  visual  display  and  into  the  audio  or  haptic 
space.  An  alert  that  an  autonomously  navigating  robot  is  stuck  and  needs  assistance  in  tele-operating  out  of  its  jam  may 
now  be  given  aurally  instead  of  visually.  The  location  of  the  robot  relative  to  the  Soldier  may  also  be  possible  through 
aural  or  haptic  cueing.  Many  variations  on  this  theme  are  possible. 

Taking  this  notion  one  step  further,  it’s  possible  for  displays  to  modify  themselves  on  the  fly,  or  be  adaptive  to  the 
operator’s  needs.  If  a  Soldier  is  controlling  an  unmanned  asset  inside  a  vehicle  from  a  crew  station,  and  then  needs  to 
utilize  the  Crew  Remote  Interface  System  (CRIS)  as  he/she  exits  the  vehicle,  it  would  be  desirable  if  the  system  would 
monitor  the  state  of  the  crew-station  interface,  and  then  automatically  configure  the  smaller  display  to  show  the  same 
things  being  monitored  on  the  larger  display  with  controls  and  displays  now  ported  across  all  available  devices  such  as  a 
PDA  device,  HMD,  and  tactile  vest.  Further  this  notion  even  one  step  more.  Say  a  fellow  Soldier  was  incapable  of 
controlling  one  of  his  assets  and  had  to  hand  off  control  to  another  Soldier  already  controlling  two  small  UGV’s.  In 
mid-mission  his  displays  could  adapt  to  this  change  and  reconfigure  themselves  using  all  available  input  and  display 
devices  to  pick-up  these  extra  tasks,  but  display  all  needed  information  in  the  optimum  configuration,  perhaps  even 
simultaneously  changing  the  level  of  automation  on  the  platforms  being  controlled.  This  synchronicity  between  the 
Soldier,  his  assets,  and  his  display  and  control  devices  is  what  the  HRI  Program  is  starting  to  address. 

5.  CONCLUSION 

The  HRI  Program  is  in  the  process  of  tackling  a  difficult  Army  problem;  providing  the  Soldier  with  the  proper  tools  and 
equipment  at  varying  mission  levels  to  enable  mixed  initiative  operations  with  unmanned  systems.  These  unmanned 
systems  will  ultimately  increase  the  stand  off  distance  with  the  enemy  and  expand  situational  awareness  of  the 
battlefield,  but  currently  cause  large  workload  demands  on  the  Soldier  and  require  unique  training  for  each  controller 
solution.  These  two  concerns  are  the  primary  focus  of  the  HRI  Program.  By  providing  a  methodology  for  normalizing 
and  scaling  missions  across  heterogeneous  system  interfaces,  TARDEC  makes  a  dramatic  improvement  in  reducing 
training  requirements  on  the  Soldier  as  he/she  interacts  with  different  unmanned  systems.  Additionally,  by  intuitively 
laying  out  the  interface  in  a  human-centric  fashion  and  providing  automations  that  perform  tasks  when  the  Soldier  is 
overloaded,  HRI  will  help  the  Soldier  realize  unparallel  levels  of  performance,  better  enable  him/her  to  focus  on  his/her 
primary  mission,  and  optimize  the  way  Soldier’s  team  with  unmanned  systems. 
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